The growth of the Internet has created limitless possibilities for information sharing and electronic commerce. Unfortunately, this growth has created commensurate opportunities for malicious intrusion. Computer network intrusions encompass deliberate attempts to access or manipulate information, to obtain services, or to render a computer system unreliable or unusable. Effective tools and methods are needed to detect intrusions as early as possible, so that effective preemptive action may be taken.
FIG. 1 is a schematic representation of the Internet 110 and several illustrative network devices that are connected by the Internet. Networked users/devices 120 communicate via message packets typically according to the Internet Transmission Control Protocol/Internet Protocol (TCP/IP). Packets contain source Internet Protocol addresses (IP addresses), destination IP addresses, and other information to connect and exchange information between the source and destination computers. Switches 130 receive and forward packets from multiple users within a local network such as network 160 according to packet header information. Routers 140 in turn interconnect two or more local networks and provides connection to the Internet.
Various security measures may be used to protect network users/devices 120. For example, network site/segment 160 connects to the Internet through router 140 and firewall 150. The firewall is placed at the connection point of the local network to the Internet. The firewall actively filters incoming and outgoing traffic, protecting against unauthorized access by computers. Network administrators can grant access to local network 160 only to selected “trusted” users or “safe” services. Outgoing traffic can also be filtered to protect valuable data. The usefulness of the firewall is limited by the need for a priori knowledge of authorized users, services and protected data. In addition, as a consequence of service interruption or other costs, it may be inconvenient to move a firewall from an existing location to a more interesting or convenient one.
Malicious activity may arise from any computer within the network or from multiple computers acting in concert. Typically, the malicious entity launches TCP/IP-based “probes,” which are attempts to connect with targeted network devices. “Scans” are systematic groups of probes originating from a single source computer or group of collaborating sources. Scans and probes are executed by malicious users and worms to find opportunities to attack or break into a targeted victim computer, and typically precede the actual attack. The attack itself may be an attempt to breach the security to the computer to obtain, e.g., user identification, access codes or other proprietary information or to interfere with the operation of the computer, e.g., by overloading resources, or redirecting the processing capabilities of the computer. All of these activities—probes, scans and attacks—are view as security threats. As will be appreciated, it is advantageous to detect probing and scanning sources to forewarn of a likely subsequent attempt to attack or break into a target victim computer.
Prior art Intrusion Detection Systems (IDSs) passively monitor network traffic for suspicious activity. As shown in FIG. 1, an IDS 170 utilizes one or more data sensors 180 attached to a network “tap point” 190 to collect and summarize critical parameters. Tap point 190 provides all traffic passing through the point in either direction. The end goal of an IDS is the reliable detection of probes and scans among other intrusive activities such as hacker attacks and break-ins to take control of a target computer. IDSs may utilize dedicated or distributed resources (i.e., resources distributed among sites in a large network).
Prior art intrusion detection methods include misuse detection and anomaly detection. Misuse detection requires a priori knowledge of an attack pattern. Online activity is evaluated with respect of a model of the malicious behavior, and activity that is consistent with the misuse model is flagged. Misuse detection offers the advantage of requiring relatively low computational resources. However, attack signatures must be known, and the misuse model must be designed to encompass all possible variations of the pertinent attack. Unfortunately, malicious users and programmers who write new worms often discover new ways to attack that are not known to programmers who write signature rules to detect attacks; as a result, IDSs often do not detect these attacks.
Anomaly detection evaluates network activity with respect to a model of normal behavior and flags inconsistent activity as anomalous. Anomaly detection systems thus offer the advantage of being able to recognize unknown attacks. In practice, the set of actual intrusive activities is not exactly the same as the set of anomalous activities, i.e., the two sets only intersect. Consequently, anomalous activities that are not intrusive may be incorrectly detected as intrusive (“false positives,” or “false alarms”), and actual intrusive activities that are not anomalous may result in events that are not flagged as intrusive (“false negatives”). Detection threshold levels must be chosen to appropriately balance the incidence of these occurrences. In general, it is desirable to maximize the probability of correct detection while regulating the false-positive rate. Anomaly detection systems generally suffer the disadvantage of being computationally expensive (i.e., with regard to CPU and memory resources).
IDSs in general face additional challenges. The asymmetrical nature of network data makes the design of intrusion detection algorithms challenging. “Stealthy” surveillance may be spread over long time spans and may therefore be camouflaged by legitimate traffic. New attack agendas are generally unknown, and deterministic analysis of raw sensor data may be impractical due to imprecise knowledge of the local network configuration
Typically, prior art IDSs have been limited to measuring the “spread” of connections from a given source, i.e., if the source connects or attempts to connect to too many destinations in a given time, or if too many connections are attempted in a given amount of time, the source is considered to be malicious. This strategy consumes an intractable quantity of memory when applied to a large network, i.e., one having a large number of network addresses. As a result, it is necessary to limit the amount of time during which statistics are gathered; and stealthy, long duration attacks or very slow probing goes undetected.
For example, SPICE/SPADE has been developed to detect portscans and other stealthy probes. It uses a probabalistic behavioral model to compute an anomaly score. This IDS suffers the disadvantage of requiring large amounts of memory to track packet distributions across combinations of source and destination IP addresses.
EMERALD from SRI International has also been used to detect portscans. It constructs statistical profiles for source IP addresses, and compares a short-term weighted behavior profile (e.g., number of SYN packets) to a long-term weighted profile. If the short-term profile deviates significantly from the long-term profile, the source is considered suspicious. This approach cannot detect slow, stealthy scans and cannot easily correlate distributed source scans.
The SNORT portscan preprocessor is an open-source IDS that looks for a specific quantity of TCP/UDP packets sent to any number of host/port combinations from a single source within a specified time duration. SNORT also looks for single TCP packets having an unusual combination of flags not normally used for TCP connections. SNORT is unable to detect scans originating from multiple hosts. Also, the quantity and duration thresholds are statically computed, making it easy for a malicious party to avoid detection by increasing the time between scanning probes.
The prior art approaches are thus incapable of automatically and effectively detecting slow, stealthy surveillance activities or new, unknown threats, or significant variations of known threats. Accordingly, there is a need for new methods and tools that can automatically detect, characterize and enable effective response to new threats without consuming inordinate computational and human resources.